Developing Mapping and Evaluation Techniques for Textual Case-Based Reasoning

Textual Case-Based Reasoning (CBR) is not simply Information Retrieval (IR) of text documents which happen also to be cases. Nor does it involve only techniques for automatically determining what cases represented as texts are about or techniques for automatically indexing such cases under relevant features. Textual CBR is still case-based reasoning, and for us, that means drawing inferences about problem situations by comparing them to past decided cases. This specific property distinguishes textual CBR from other techniques such as IR. In other words, we believe that (1) textual CBR should also involve drawing inferences by computationally comparing cases expressed as texts. More accurately, given the technical difficulties of computationally comparing case texts, we take this as our working hypothesis. We believe it is worth investing a great deal of effort to confirm this hypothesis before we are willing to accept a watered-down conception of what textual CBR should be about. At the same time, we do not minimize the difficulties of computationally comparing case texts. While cases may be compared computationally in terms of numerical feature weights, symbolic arguments, or adaptations, arranging for cases described as texts to be compared in these ways presents many research questions. Chief among the problems to be solved is mapping text to comparison-enabling structures. We maintain that (2) to be successful, textual CBR will require textual descriptions of cases to be mapped onto structural representations which facilitate computationally comparing cases. It is important to realize that textual CBR does not avoid the need for structured case comparisons. While it may be true, as the Call for Participation asserts, that "many CBR applications now require the handling of only semi-structured or even full-text cases rather than the highly structured cases of more traditional CBR systems", textual CBR cannot dispense with structure in so far as it is necessary to support case comparison. In this paper, we elaborate one such structure called "factors" which facilitates comparing problems and cases. Our experiments in automatically classifying cases expressed as texts are aimed at identifying the factors implicit in the textual case descriptions. Evaluation is a crucial step in any research project, and evaluating textual CBR systems is particularly challenging. We maintain that (3) an evaluation of textual CBR has to be designed carefully to disentangle the aspects to be assessed from other aspects of what is inevitably a complex system and to adopt appropriate standards for comparing system performance. Moreover, evaluation criteria for measuring success must augment those commonly applied to assess information retrieval of textual documents. The widelyused IR measures precision/recall do not capture all of the information required for assessing the performance of a textual CBR system. We develop this position on evaluation in a separate position paper.